GUITest: a Java library for fully automated GUI robustness testing

© Sebastian Bauersfeld, Tanja E. J. Vos | ACM 2012. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ASE 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, http://dx.doi.org/10.1145/2351676.2351739Graphical User Interfaces (GUIs) are substantial parts of today’s applications, no matter whether these run on tablets, smartphones or desktop platforms. Since the GUI is often the only component that humans interact with, it demands for thorough testing to ensure an efficient and satisfactory user experience. Being the glue between almost all of an application’s components, GUIs also lend themselves for system level testing. However, GUI testing is inherently diffi- cult and often involves great manual labor, even with modern tools which promise automation. This paper introduces a Java library called GUITest, which allows to generate fully automated GUI robustness tests for complex applications, without the need to manually generate models or input sequences. We will explain how it operates and present first results on its applicability and effectivity during a test involving Microsoft Word.This work is supported by EU grant ICT-257574 (FITTEST).Bauersfeld, S.; Vos, TE. (2012). GUITest: a Java library for fully automated GUI robustness testing. En ASE 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. ACM. 330-333. https://doi.org/10.1145/2351676.2351739S33033

Enseñanza temprana del testing en cursos de programación

Vos, TE.; Marín, B. (2021). Enseñanza temprana del testing en cursos de programación. Escola Tècnica Superior d'Enginyeria Informàtica. 60-66. http://hdl.handle.net/10251/177475S606

A Methodological Framework for Evaluating Software Testing Techniques and Tools

© 2012 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.There exists a real need in industry to have guidelines on what testing techniques use for different testing objectives, and how usable (effective, efficient, satisfactory) these techniques are. Up to date, these guidelines do not exist. Such guidelines could be obtained by doing secondary studies on a body of evidence consisting of case studies evaluating and comparing testing techniques and tools. However, such a body of evidence is also lacking. In this paper, we will make a first step towards creating such body of evidence by defining a general methodological evaluation framework that can simplify the design of case studies for comparing software testing tools, and make the results more precise, reliable, and easy to compare. Using this framework, (1) software testing practitioners can more easily define case studies through an instantiation of the framework, (2) results can be better compared since they are all executed according to a similar design, (3) the gap in existing work on methodological evaluation frameworks will be narrowed, and (4) a body of evidence will be initiated. By means of validating the framework, we will present successful applications of this methodological framework to various case studies for evaluating testing tools in an industrial environment with real objects and real subjects.This work was funded by the European project FITTEST (ICT257574, 2010-2013) and Spanish National project CaSA-Calidad (TIN2010-12312-E, Ministerio de Ciencia e Innovación)Vos, TE.; Marín, B.; Escalona, MJ.; Marchetto, A. (2012). A Methodological Framework for Evaluating Software Testing Techniques and Tools. IEEE. https://doi.org/10.1109/QSIC.2012.16

Evaluating the use of self-video teaching in a flipped classroom

[EN] New generations are increasingly becoming more familiar with consuming audio-visual material through online platforms. Consequently, learning through IT-based tools is becoming more and more common. Nowadays, learning platforms (e.g., edX or W3Schools) or content platforms (e.g., YouTube) containing vast amounts of courses and video tutorials are becoming increasingly popular among students. The main advantage of online learning is that students can access the content from anywhere and whenever they want, being able to revisit the content to review concepts and improve their level of knowledge. In this way, learning based on a deep approach and self-learning is promoted since students are the ones who regulate their learning process by deciding how much time to dedicate and when to do it. Appropriately using this type of resource can become a very effective tool applied to a flipped classroom model. In the flipped classroom model, students are active learners since they are in charge of developing the lesson material both in class and at home. In this type of learning, the teacher assumes the role of guide assisting during the learning process. A standard methodology in this flipped classroom model consists of students preparing different parts of the course content and then explaining those parts to their classmates. In this way, students develop a sense of responsibility toward the rest of their classmates, creating an environment where they can recognise their shortcomings and take control of their learning to teach others. In addition, the acquisition of transversal communication skills is encouraged. With all this in mind, in this article, we describe a case study we are currently carrying out with students enrolled in the programming course at the Universitat Politècnica de València. Our proposal combines the flipped classroom model with access to online resources. In this first approach, we have proposed that the students record a video explaining a part of the lesson or how to solve at least two exercises step by step. The explanation must be done as if they were content creators, and their audience were beginner programmers. The students will upload the videos to a private YouTube channel that will only be accessible to their classmates. In the classroom, the teacher will encourage students to share their stories and experiences while learning, editing, and recording the videos. This proposal's main objective is to promote students' engagement in the learning process and offer them learning alternatives through online content with a closer language that they can access whenever they need it. To motivate participation, students and teachers will choose the three best videos from all the videos. The three winners will receive extra points in the evaluation of the course.The authors gratefully acknowledge the financial support of Consellería d'Innovació, Universitats, Ciencia i Societat Digital from Comunitat Valenciana and the European Social Fund (Investing In Your Future) (APOSTD/2021/227 and CIPROM/2021/077), the Spanish Ministry of Science (project PID2021-123673OB-C31) and the Research Services of Universitat Politècnica de València. Jaume Jordán is supported by grant IJC2020-045683-I funded by MCIN/AEI/10.13039/501100011033 and by "European Union NextGenerationEU/PRTR".Taverner-Aparicio, JJ.; Marco-Detchart, C.; Jordán, J.; Vos, TE. (2023). Evaluating the use of self-video teaching in a flipped classroom. IATED. 3133-3138. https://doi.org/10.21125/inted.2023.08653133313

DECODER - DEveloper COmpanion for Documented and annotatEd code Reference

This work has been developed with the financial support of the European Union's Horizon 2020 research and innovation programme under grant agreement No. 824231Gil Pascual, M.; Pastor-Ricós, F.; Torres Bosch, MV.; Vos, TE. (2020). DECODER - DEveloper COmpanion for Documented and annotatEd code Reference. Springer. 643-644. http://hdl.handle.net/10251/178910S64364

Using genetic programming to evolve action selection rules in traversal-based automated software testing: results obtained with the TESTAR tool

[EN] Traversal-based automated software testing involves testing an application via its graphical user interface (GUI) and thereby taking the user's point of view and executing actions in a human-like manner. These actions are decided on the fly, as the software under test (SUT) is being run, as opposed to being set up in the form of a sequence prior to the testing, a sequence that is then used to exercise the SUT. In practice, random choice is commonly used to decide which action to execute at each state (a procedure commonly referred to as monkey testing), but a number of alternative mechanisms have also been proposed in the literature. Here we propose using genetic programming (GP) to evolve such an action selection strategy, defined as a list of IF-THEN rules. Genetic programming has proved to be suited for evolving all sorts of programs, and rules in particular, provided adequate primitives (functions and terminals) are defined. These primitives must aim to extract the most relevant information from the SUT and the dynamics of the testing process. We introduce a number of such primitives suited to the problem at hand and evaluate their usefulness based on various metrics. We carry out experiments and compare the results with those obtained by random selection and also by Q-learning, a reinforcement learning technique. Three applications are used as Software Under Test (SUT) in the experiments. The analysis shows the potential of GP to evolve action selection strategies.Esparcia Alcázar, AI.; Almenar-Pedrós, F.; Vos, TE.; Rueda Molina, U. (2018). Using genetic programming to evolve action selection rules in traversal-based automated software testing: results obtained with the TESTAR tool. Memetic Computing. 10(3):257-265. https://doi.org/10.1007/s12293-018-0263-8S257265103Aho P, Menz N, Rty T (2013) Dynamic reverse engineering of GUI models for testing. In: Proceedings of 2013 international conference on control, decision and information technologies (CoDIT’13)Aho P, Oliveira R, Algroth E, Vos T (2016) Evolution of automated testing of software systems through graphical user interface. In: Procs. of the 1st international conference on advances in computation, communications and services (ACCSE 2016), Valencia, pp 16–21Alegroth E, Feldt R, Ryrholm L (2014) Visual GUI testing in practice: challenges, problems and limitations. Empir Softw Eng 20:694–744. https://doi.org/10.1007/s10664-013-9293-5Barr ET, Harman M, McMinn P, Shahbaz M, Yoo S (2015) The oracle problem in software testing: a survey. IEEE Trans Softw Eng 41(5):507–525Bauersfeld S, Vos TEJ (2012) A reinforcement learning approach to automated GUI robustness testing. In: Fast abstracts of the 4th symposium on search-based software engineering (SSBSE 2012), pp 7–12Bauersfeld S, de Rojas A, Vos T (2014) Evaluating rogue user testing in industry: an experience report. In: 2014 IEEE eighth international conference on research challenges in information science (RCIS), pp 1–10. https://doi.org/10.1109/RCIS.2014.6861051Bauersfeld S, Vos TEJ, Condori-Fernández N, Bagnato A, Brosse E (2014) Evaluating the TESTAR tool in an industrial case study. In: 2014 ACM-IEEE international symposium on empirical software engineering and measurement, ESEM 2014, Torino, Italy, September 18–19, 2014, p 4Bauersfeld S, Wappler S, Wegener J (2011) A metaheuristic approach to test sequence generation for applications with a GUI. In: Cohen MB, Ó Cinnéide M (eds) Search based software engineering: third international symposium, SSBSE 2011, Szeged, Hungary, September 10-12, 2011. Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 173–187Brameier MF, Banzhaf W (2010) Linear genetic programming, 1st edn. Springer, New YorkChaudhary N, Sangwan O (2016) Metrics for event driven software. Int J Adv Comput Sci Appl 7(1):85–89Esparcia-Alcázar AI, Almenar F, Martínez M, Rueda U, Vos TE (2016) Q-learning strategies for action selection in the TESTAR automated testing tool. In: Proceedings of META 2016 6th international conference on metaheuristics and nature inspired computing, pp 174–180Esparcia-Alcázar AI, Almenar F, Rueda U, Vos TEJ (2017) Evolving rules for action selection in automated testing via genetic programming–a first approach. In: Squillero G, Sim K (eds) Applications of evolutionary computation: 20th European conference, evoapplications 2017, Amsterdam, The Netherlands, April 19–21, 2017, Proceedings, part II. Springer, pp 82–95. https://doi.org/10.1007/978-3-319-55792-2_6Esparcia-Alcázar AI, Moravec J (2013) Fitness approximation for bot evolution in genetic programming. Soft Comput 17(8):1479–1487. https://doi.org/10.1007/s00500-012-0965-7He W, Zhao R, Zhu Q (2015) Integrating evolutionary testing with reinforcement learning for automated test generation of object-oriented software. Chin J Electron 24(1):38–45Koza JR (1992) Genetic programming: on the programming of computers by means of natural selection. MIT Press, CambridgeLehman J, Stanley KO (2011) Novelty search and the problem with objectives. In: Riolo R, Vladislavleva E, Moore JH (eds) Genetic programming theory and practice IX, genetic and evolutionary computation. Springer, New York, pp 37–56Memon AM, Soffa ML, Pollack ME (2001) Coverage criteria for GUI testing. In: Proceedings of ESEC/FSE 2001, pp 256–267Rueda U, Vos TEJ, Almenar F, Martínez MO, Esparcia-Alcázar AI (2015) TESTAR: from academic prototype towards an industry-ready tool for automated testing at the user interface level. In: Canos JH, Gonzalez Harbour M (eds) Actas de las XX Jornadas de Ingeniería del Software y Bases de Datos (JISBD 2015), pp 236–245Seesing A, Gross HG (2006) A genetic programming approach to automated test generation for object-oriented software. Int Trans Syst Sci Appl 1(2):127–134Vos TE, Kruse PM, Condori-Fernández N, Bauersfeld S, Wegener J (2015) TESTAR: tool support for test automation at the user interface level. Int J Inf Syst Model Des 6(3):46–83. https://doi.org/10.4018/IJISMD.2015070103Wappler S, Wegener J (2006) Evolutionary unit testing of object-oriented software using strongly-typed genetic programming. In: Proceedings of the 8th annual conference on genetic and evolutionary computation, GECCO’06. ACM, New York, NY, USA, pp 1925–1932. URL https://doi.org/10.1145/1143997.1144317Watkins C (1989) Learning from delayed rewards. Ph.D. Thesis. Cambridge Universit

Evaluating TESTAR's effectiveness through code coverage

[EN] Testing is of paramount importance in assuring the quality of software products. Nevertheless, it is not easy to judge which techniques or tools are the most effective. A commonly used surrogate metric to evaluate the effectiveness of testing tools is code coverage, which has been widely used for unit and integration testing. However, for GUI testing approaches, this metric has not been sufficiently investigated. To fill this gap, we run experiments with the TESTAR tool, a scriptless testing tool that automatically generates test cases at the Graphical User Interface (GUI) level. In the experiment, we analyze and compare the obtained code coverage when using four different action selection mechanisms (ASMs) in TESTAR that are used to test three SUTsThis research has been funded by the following projects: H2020 EU project DECODER (www.decoder-project.eu), H2020 EU project iv4XR (www.iv4xrproject.eu) and ITEA project IVVES (www.ivves.eu).Van Der Brugge, A.; Pastor-Ricós, F.; Aho, P.; Marín, B.; Vos, TE. (2021). Evaluating TESTAR's effectiveness through code coverage. SISTEDES. 1-14. http://hdl.handle.net/10251/178270S11

Applying Scriptless Test Automation on Web Applications from the Financial Sector

[EN] This industry showcase presents experiences on application of testar, an open source tool for scriptless testing through graphical user interface (GUI), to the web applications of Kuveyt T¿urk Participation Bank in Turkey. Kuveyt T¿urk Bank uses Selenium and Appium for regression testing of mobile and internet banking, but the maintenance cost of the test scripts is increasing day by day. Therefore, scriptless GUI testing with testar was evaluated. To provide better support for testing web-based applications, testar was extended with Selenium WebDriver integration, JavaScript support, and other new features. Results show that testar detects GUI elements much better after the improvements, and it was able to find 2 relevant errors that were not identified by existing scripted test cases.This work has been partially funded by ITEA3 TESTOMAT Project, ITEA3 IVVES project7 and EU H2020 DECODER projecAho, P.; Buijs, G.; Akin, A.; Senturk, S.; Pastor-Ricós, F.; De Gouw, S.; Vos, TE. (2021). Applying Scriptless Test Automation on Web Applications from the Financial Sector. SISTEDES. 1-4. http://hdl.handle.net/10251/178254S1

Future Internet Testing. FITTEST

This is a video that presents a high level overview of the FITTEST project for Future Internet Testing. What is it, Who is participating, Why is it needed, What is delivered, Who can benefit and How can you participate.https://polimedia.upv.es/visor/?id=cd34a9f1-dfcb-0d4d-b84d-0ef6f8a43228Vos, TE. (2013). Future Internet Testing. FITTEST. http://hdl.handle.net/10251/3015

Testar. Test Automation at the user interface level

TESTAR, a tool for test automation at the user interface levelhttps://polimedia.upv.es/visor/?id=bc56b785-9f0e-6743-b3bd-9f159fa16fe0Vos, TE. (2014). Testar. Test Automation at the user interface level. http://hdl.handle.net/10251/3840